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200 



FOR A SESSION, OBTAIN 
MODALITY SPECIFIC 
INSTRUCTIONS FOR A 
PLURALITY OF USER AGENT 
PROGRAMS THAT OPERATE IN 
DIFFERENT MODALITIES 
WRT EACH OTHER 



202 -v 



ON A PER SESSION BASIS, 
SYNCHRONIZE OUTPUT BY THE 
PLURALITY OF USER AGENT 
PROGRAMS BASED ON THE 
MODALITY SPECIFIC 
INSTRUCTIONS TO FACILITATE 
CONCURRENT MULTIMODAL 
OPERATION FOR A USER 
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DO SET OF 
MODALITY SPECIFIC 
INSTRUCTIONS FOR DIFFERENT 
USER AGENT PROGRAMS REQUEST 
CONCURRENT INPUT OF INFORMATION IN DIFFERENT 
MODALITIES BY A USER THROUGH 
THE DIFFERENT USER 
AGENT PROGRAMS 



FORWARD RECEIVED INPUT 
INFORMATION FROM ONE USER 
AGENT PROGRAM TO SERVER 
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.YES 



FUSE RECEIVED CONCURRENT 
DIFFERENT MULTIMODAL INPUT 
INFORMATION SENT FROM THE 
AT LEAST TWO USER AGENT 
PROGRAMS TO PRODUCE A 
FUSED MM RESPONSE 
ASSOCIATED WITH DIFFERENT 
USER AGENT PROGRAMS 
OPERATING IN DIFFERENT 
MODALITIES 
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FORWARD THE FUSED MM 
RESPONSE TO ONE OR 
MORE APPLICATIONS 
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( START ) 
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SEND REQUEST FOR WEB CONTENT OR 
OTHER INFORMATION VIA A FIRST USER 
AGENT PROGRAM (E.G., DEVICE SENDS 
CONTENT REQUEST TO WEB SERVER 
THROUGH BROWSER) 



302 



~X 



I 




CONTENT SERVER OBTAINS MM 
PREFERENCE DATA FROM MODALITY 
PREFERENCE DATABASE FOR IDENTIFIED 
USER TO OBTAIN DEVICE PREFERENCE 
AND MODE PREFERENCE FOR SESSION 
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CONTENT SERVER NOTIFIES MULTIMODAL 

FUSION SERVER WHICH USER AGENT 
APPLICATIONS ARE OPERATING ON WHICH 
DEVICES AND IN WHICH MODE FOR THE 
CONCURRENT MM COMMUNICATION 
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SET UP MM SYNC COORD TO DETERMINE 
RESPECTIVE PROXIES FOR EACH 
DIFFERENT MODALITY BASED ON 
MODALITY PREFERENCE INFORMATION 
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RECEIVE USER MODE SELECTION FOR 
EACH USER AGENT PROGRAM 



310 



I 



FETCH AND TRANSCODE MODALITY 
SPECIFIC INSTRUCTIONS FROM CONTENT 
SERVER FOR EACH USER AGENT 
PROGRAM (I.E., FOR EACH MODALITY) 
BASED ON ACTIVE MODALITIES FOR 
SESSION 
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RESET PROXIES IF USER 
DESIGNATES CHANGES 

I 



REACH LISTENING POINT 
(END OF STATE) 
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SYNCHRONIZE MODALITY SPECIFIC 
INSTRUCTIONS (E.G. MARKUP LANGUAGE 
FORMS) FOR DIFFERING USER AGENT 
PROGRAMS (E.G. DIFFERENT MODES) FOR 
CONCURRENT REQUESTS IN DIFFERING 
MODES TO DIFFERING USER 
AGENT PROGRAMS 
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SEND, VIA CORRESPONDING PROXIES, 
SYNCHRONIZED MODALITY SPECIFIC 
INSTRUCTIONS TO REQUEST USER INPUT 
INFORMATION IN DIFFERING MODALITIES 
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ERROR 



FUSE RECEIVED 

INPUT 
INFORMATION 
FROM USER 
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( START ) 



700 



MAINTAINING, DURING NON-SESSION 
CONDITIONS ON A PER USER BASIS. 
MULTIMODAL SESSION STATUS 
INFORMATION OF USER AGENT 
PROGRAMS CONFIGURED FOR DIFFERENT 
CONCURRENT MODALITY COMMUNICATION 
DURING A SESSION 
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702 



704 



DURING A CONCURRENT MM SESSION, 
STORE PER USER MM SESSION STATUS 
INFORMATION (E.G., USER ID 
DATA, SESSION ID DATA, ID 
DATA FOR EACH USER AGENT PROGRAM 
OPERATING CONCURRENTLY IN 
DIFFERENT MODALITIES DURING THE 
SESSION, AND STATE INFORMATION 
OF EACH USER AGENT PROGRAM 



RE-ESTABLISHING A CONCURRENT 
MULTIMODAL SESSION IN RESPONSE TO 
ACCESS OF THE MULTIMODAL SESSION 
STATUS INFORMATION 



FIG. 6 
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DETECT JOINING OF A 
SESSION BY A USER 
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ACCESS STORED MULTIMODAL SESSION 
STATUS INFORMATION IN MEMORY BASED 
ON DETECTION OF JOINED SESSION 



NO 



DOES \^ 710 

session exist 
in concurrent mm session 
status memory 

'yes 



QUERY MEMORY \^ 7 14 
TO SEE IF USER 
HAS EXISTING APPLICATION \ N0 
RUNNING AND IF SO IF USER WOULD 
LIKE TO RE-ESTABLISH 
COMMUNICATION WITH 
JHE APPLICATION 
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YES 
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RETRIEVE REDIRECT (RE-ESTABLISH) URL 
716 H FROM CONCURRENT MM SESSION 

MEMORY 
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PROVI0E REDIRECT URL FOR 
RESPECTIVE MM PROXY 



SEND REQUEST TO APPROPRIATE USER 
AGENT PROGRAM, VIA PROXY, BASED ON 
72£H STORED USER AGENT STATE 
INFORMATION FROM CONCURRENT 
MM SESSION MEMORY 
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